MA Bobulsky, Louis Stokes Cleveland Department of Veterans Affairs Medical Center; J Spencer, Boston VA Health Care Center; WG Henderson, Hines Center for Cooperative Studies in Health; J Daley, MGH/Partners Health Care System; S Khuri, Boston VA Health Care Center

Objectives: To develop a cost-effective system for measuring Surgical Clinical Nurse Reviewer's, (SCNR), competency and inter-rater reliability, (IRR), in completion of risk assessments for the National Surgical Quality Improvement Program, (NSQIP). The Veterans Health Administration's NSQIP is a national, validated, outcome-based, risk-adjusted, and peer-controlled program for the measurement and enhancement of surgical quality of care. A SCNR collects, validates, transmits and maintains NSQIP data at each participating VAMC. The SCNR utilizes detailed definitions of each pre-operative, operative and outcome measure in completing the risk assessments. While internal validity checks are built into the database, site specific IRR testing had not been conducted since 1993. At that time IRR was ascertained by two traveling nurse coordinators who visited each of 44 VAMCs and conducted an independent abstraction of the data. Nursing competency focuses on one's ability to integrate knowledge and skills in performance. Verifying a SCNR's competency ensures that the SCNR can actually complete the risk assessment as expected. This provides the measure of the SCNR's competency. Since the program has now expanded to include 123 VAMCs, the need has risen for a more practical and cost-effective system of ascertaining both IRR and nurse competency.

Methods: The ten NSQIP Regional Nurse Coordinators, (each supervising the SCNRs at 10-13 VAMCs), were divided into two groups. Group 1 developed an educational competency module to assess the SCNR's ability to complete a risk assessment according to NSQIP criteria. This module was sent electronically or by mail to each NSQIP site to be completed by all individuals who collect NSQIP data.

Group 2 developed a case study module to assess SCNR IRR. The case study was electronically sent on the same day to all sites. The SCNRs in the field logged onto a national conference call to confirm receipt of the case study. Each SCNR completed the risk assessment for this case study and faxed it to their respective Regional Coordinator by COB.

Results: All SCNRs, (100%), completed the competency module and IRR case study examination. The mean SCNR scores for IRR and competency were 98.8% and 95.1% respectively. These scores demonstrate excellent IRR and competency. The module was designed so that the IRR and SCNR competency could be assessed periodically at minimal cost, (i.e. the cost of a national teleconference call). The alternative of assessing IRR through individual site visits would exceed $100,000 in cost.

Conclusions: The validity and reliability of the NSQIP database coupled with the assurance that competent SCNRs are collecting the data are crucial elements to the accuracy of the NSQIP data collection. This process was less time-consuming and more cost-effective than the IRR site visit method previously used.

Impact: Electronic dissemination of testing materials, scoring and feedback provide the opportunity to conduct continuous monitoring and improvement in data collection in a national health care system.